Multi-frame integration in video (image sequence) processing is an important and useful technology that has drawn much attention recently. One user case and application example is multi-frame super-resolution (EP 09150632.9, filed 15 Jan. 2009) in which a frame with improved resolution is created from a sequence of frames combined. Another one is conversion of conventional videos to high definition, HD, videos.
Estimation of the inter-frame motion fields is an essential part in multi-frame integration, and as a matter of fact the most time-consuming part. When video output with a high enough rate (e.g., around 30 frames per second) is required, the multi-frame integration process has to be done recursively at a similar rate. This puts a tight constraint on the motion field estimation stage.
This invention addresses among other things how to speed up the recursion loop and proposes a solution to the acceleration of the motion field estimation loop.
FIG. 1 illustrates schematically how one video output multi-frame super-resolution (Mf-SR) frame is generated in the prior art. N frames of the input video sequence are to be integrated over a Time of Integration (TOI) window to deliver each frame of the output video in Mf-SR. Within the TOI window, one frame is selected as the reference frame, here the first one #0, but may also be e.g. the last one #N−1. Then the inter-frame motion fields of all the rest frames with respect to the chosen reference frame are computed. A motion field relating to two frames consists of a vector for each pixel pointing from one pixel in one frame to the corresponding pixel in the other frame, in other words representing the motion of the pixel between the two frames. With the known motion fields, all the images within the TOI window are aligned and a high resolution image (as one output video frame in Mf-SR) from N frames of the low resolution input video is generated, as suggested in EP 09150632.9. For the initial TOI window consisting of frames #0 to N−1, the first output frame in Mf-SR is generated. For next TOI window frames #1 to N are used etc.
The motion vectors between two frames can be estimated by one of the conventional methods e.g. as described in the article by L. Barron, D. J. Fleet, and S. S. Beauchemin, Performance of optical flow techniques, International Journal of Computer Vision, 12(1):43-77, 1994.
Image pair-wise motion fields Mr,j between the reference frame r and other target frame j (r<j<r+N) need to be estimated. One implementation would be using the first frame of the TOI as the reference frame (FIG. 1). With frame 0 being the reference frame, we need to estimate M0,1, M0,2, . . . , M0,N−1. As the recursion proceeds with the next TOI window, another frame r in the input video sequence is taken as the reference frame later on. Then we need Mr,r+1, Mr,r+2, . . . , Mr,r+N−1. This means that in general we have to estimate N−1 motion fields within each TOI window.
An alternative implementation is using the last frame of the TOI as the reference frame (not shown). In this case, frame N−1 is the reference one, and similar to the previous case, we need to compute N−1 new motion fields within one TOI window or each time the reference frame is changed as the TOI window moves along the time axis.
There is one main problem with this method. The motion (vector) field of each image within the TOI is to be estimated with respect to the reference image (either the first or the last image of the TOI), with a total of N−1 motion fields. This is referred to as absolute motion estimation within the TOI window. As the recursion goes on, the TOI window shifts along the time axis and the reference image needs to be re-defined. Thus, the motion fields of all the images that remain in the new TOI have to be re-computed (with respect to the new reference image), which is a large computation burden.